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Executive  Summary 

This  report  describes  a set  of  performance  metrics  and  a performance  measurement  system 
developed  for  the  evaluation  of  roadway  departure  warning  systems.  The  work  is  part  of  an 
effort  by  the  Department  of  Transportation’s  National  Highway  Traffic  and  Safety 
Administration  (NHTSA)  to  develop  quantitative  measures  of  performance  for  the  evaluation  of 
highway  safety  systems.  The  work  is  also  part  of  an  effort  to  promote  dual-use  technologies 
between  the  Department  of  Transportation  and  the  Department  of  Defense.  The  Department  of 
Commerce’s  National  Institute  of  Standards  and  Technology  undertook  the  development  of  the 
performance  measurement  system. 


The  approach  for  quantitative  evaluation  rests  on  establishing  a baseline  measurement  of 
roadway  departure  that  is  of  higher  accuracy  than  the  warning  system  under  test.  The  output  of 
the  warning  system  is  then  compared  to  the  baseline  measurement  to  produce  a quantitative 
measure  of  performance.  The  baseline  measurement  system  is  installed  on  a vehicle  with  the 
warning  system.  The  baseline  system  achieves  high  accuracy  by  aiming  a calibrated  camera  onto 
an  area  of  the  road  adjacent  to  the  vehicle.  As  the  vehicle  is  driven  and  roadway  departure 
maneuvers  are  performed,  the  output  of  the  video  camera,  the  vehicle’s  position  (from  a global 
positioning  system  and  inertial  sensors),  and  the  output  of  the  warning  system  are  synchronized 
and  digitally  recorded.  After  a vehicle  run  is  complete,  a suite  of  software  is  used  to 
automatically  analyze  the  baseline  video  data  to  determine  roadway  departure.  Since  the  video 
and  other  data  remain  available  after  the  vehicle  run,  an  operator  (to  establish  the  credibility  of 
the  baseline  measurement)  can  verify  a roadway  departure.  Also,  the  data  can  be  analyzed  to 
provide  insight  into  the  cause  of  a warning  system  failure  (e.g.,  insufficient  roadway  markings, 
high  lateral  velocity,  etc.).  The  time  of  departure  is  compared  with  the  warning  system  output  to 
determine  the  reliability,  failure  and  false  alarm  rates  of  the  warning  system. 


This  report  describes  the  performance  metrics  and  the  baseline  measurement  system.  It  also 
describes  efforts  to  determine  the  accuracy  of  the  baseline  system  in  determining  the  time  and 
position  of  roadway  departures.  In  addition,  the  precise,  synchronized  recording  of  video  and 
GPS  data  proved  useful  in  determining  the  accuracy  and  the  delays  of  GPS  when  measuring  the 
position  of  a moving  vehicle.  The  baseline  measurement  system  is  now  sufficiently  mature  to  be 
used  to  evaluate  roadway  departure  warning  systems.  The  measurement  system  also  serves  as  a 
promising  platform  to  perform  a wide  range  of  other  evaluations. 


' This  paper  was  prepared  by  U.S.  Government  employees  and  is  not  subject  to  copyright.  The  use  of  commercial 
equipment  or  products  is  for  demonstration  only  and  does  not  imply  recommendation  or  endorsement  by  NIST. 
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1 Introduction 


One  of  the  goals  of  the  Department  of  Transportation,  National  Highway  Traffic  and  Safety 
Administration  (NHTSA)  is  to  improve  safety  on  U.S.  highways.  An  avenue  for  improved  safety 
is  the  use  of  driver  warning  systems  for  collision  avoidance  and  roadway  departure.  Warning 
systems  for  roadway  departure  rely  on  advanced  sensors  and  processing  algorithms  to  warn  the 
driver  when  the  vehicle  is  leaving  the  road  unintentionally.  The  U.S.  Department  of  Defense, 
under  the  AUTONAV  (Autonomous  Vehicle  Navigation  Control  System)  project,  is  looking  at 
similar  sensors  and  algorithms  for  controlling  autonomous  ground  vehicles.  The  basis  for  the 
interaction  between  DOT  and  DOD  is  to  promote  dual-use  technologies  for  the  improvement  of 
surface  transportation  safety.  The  Army  Research  Lab  (ARL)  has  provided  the  National  Institute 
of  Standards  and  Technology  (NIST),  the  primary  research  participant,  with  a vehicle  and 
various  sensing  and  control  components  (see  Appendix  A:  Autonav  Program  Description)  to 
serve  as  the  test-bed  to  evaluate  warning  systems  and  to  develop  methods  for  performance 
measurement.  This  report  describes  a set  of  performance  metrics  and  a performance 
measurement  system  developed  for  the  evaluation  of  roadway  departure  warning  systems. 


A primary  function  of  a roadway  departure  warning  system  is  to  detect  the  location  of  the 
roadway  boundaries.  The  location  of  the  roadway  must  be  detected  sufficiently  forward  of  the 
vehicle  to  provide  time  for  both  the  warning  system  to  determine  whether  the  vehicle  is  on  a 
course  to  leave  the  lane,  and  for  the  driver  to  take  the  appropriate  action.  The  warning  system 
must  operate  on  a wide  variety  of  roads,  day  or  night,  and  under  a wide  range  of  weather 
conditions.  The  main  effort  in  the  AUTONAV/DOT  project  was  to  develop  a method  for 
quantitatively  measuring  the  performance  of  the  warning  system  under  these  varied  conditions. 
The  primary  measure  of  performance  is  the  reliability  of  the  warning  system  to  detect  roadway 
departure  incidents  and  to  reject  false  alarms.  The  method  uses  a baseline  measurement  system 
(BMS)  to  obtain  a baseline  of  how  a warning  system  is  expected  to  perform.  The  baseline  is 
compared  to  the  warning  system’s  actual  performance  in  order  to  determine  a measure  of 
reliability.  Section  2 presents  a scenario  of  how  a warning  system  is  evaluated  using  the  BMS. 
The  components  of  the  BMS  are  described  in  section  3.  Several  tests  were  conducted  to  validate 
the  measurement  capabilities  of  the  BMS.  The  results  of  the  BMS  tests  are  presented  in  section 
4. 


The  BMS  was  developed  based  on  NIST’s  experience'  with  systems  that  have  demonstrated  the 
capability  to  perform  lane  tracking  and  that  could  serve  as  a potential  warning  system.  During 
the  time  of  the  work,  there  were  no  commercially  available  systems,  but  recent  announcements 
have  been  made  that  products  will  soon  be  available.  The  final  method  for  evaluation  of  these 
warning  systems  will  depend  on  the  goals  of  the  manufacturer  and  of  the  evaluators.  For 
example,  the  evaluation  may  also  seek  to  obtain  the  response  of  the  driver  to  the  warning 
system’s  operation.  The  intent  of  the  efforts  described  in  this  report  is  to  build  an  understanding 
of  the  types  of  measurements  that  can  be  performed  and  to  demonstrate  a method  of  evaluation. 
More  work  needs  to  be  done  to  refine  the  measures  of  performance,  to  increase  the  number  of 
measurement  capabilities,  and  to  improve  the  accuracy  of  the  measurements.  The  authors 
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express  gratitude  to  Alhad  Chande  and  August  Burgett  of  the  National  Highway  Traffic  Safety 
Administration’s  Office  of  Crash  Avoidance  Research  for  their  insights,  perseverance  and 
contributions  toward  the  completion  of  this  project  and  the  report 


2 Scenario  for  a Roadway  Departure  System  Evaluation 

Several  types  of  evaluations  can  be  conducted  to  obtain  an  overall  performance  evaluation  of  a 
roadway  departure  warning  system.  A preliminary  evaluation  is  usually  conducted  prior  to  a 
field  operational  test  in  order  to  determine  the  safety  and  reliability  of  the  warning  system  under 
a controlled  set  of  circumstances.  The  results  of  a preliminary  evaluation  are  used  to  design  and 
implement  a full-scale  field  operational  test  involving  a large  number  of  drivers  operating  under 
normal  everyday  driving  conditions.  A scenario  for  a preliminary  evaluation  of  a roadway 
departure  warning  system  is  described  in  this  section.  This  scenario  drove  the  design  and 
implementation  of  a performance  measurement  system  called  the  baseline  measurement  system 
(BMS).  The  following  list  highlights  some  general  attributes  of  a preliminary  evaluation. 

1.  Ability  to  generate  a large  number  of  roadway  departure  incidents. 

2.  Ability  to  record  the  response  of  the  warning  system  to  each  incident. 

3.  Ability  to  establish  a baseline,  that  is,  the  correct  operation  expected  from  the  warning 
system,  and  to  compare  the  output  of  the  warning  system  against  the  baseline  in  a 
quantitative  manner. 

4.  Ability  to  perform  evaluations  on  real  roads  - not  just  on  a closed  test  tracks. 

5.  Ability  to  perform  evaluations  in  wide  variety  of  environmental  conditions:  lighting,  weather, 
etc. 

6.  Ability  to  perform  the  evaluations  in  a cost-effective  manner. 

The  first  step  of  an  evaluation  is  to  instrument  the  vehicle  with  the  warning  system  and  the  BMS. 
An  overview  of  the  vehicle  instrumentation  is  given  in  section  2.1.  Once  the  instrumentation  is 
in  place  and  calibrated,  the  remaining  procedures  in  an  evaluation  plan  consist  of  a series  of  test- 
drives  and  data  collections  described  in  section  2.2.  Section  2.3  describes  how  the  data  is 
analyzed  to  produce  various  measures  of  performance  that  describes  the  reliability  of  the  warning 
system. 
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Figure  1.  Vehicle  instrumentation  for  evaluating  performance  of  a roadway  departure 
warning  system. 

2.1  Vehicle  Instrumentation 

The  first  step  to  a preliminary  evaluation  of  a roadway  departure  warning  system  is  to  install  the 
warning  system  and  the  baseline  measurement  system  on  the  vehicle.  For  scenario  purposes,  it  is 
assumed  that  the  warning  system  is  vision-based,  and  the  manufacturer  installs  the  system  on  the 
vehicle  and  performs  any  required  calibrations.  Figure  1 shows  the  components  of  the  BMS  that 
are  installed  on  the  vehicle.  The  Video/Digital  Recording  subsystem  consists  of  a calibrated 
camera  and  a digital  recorder.  The  camera  is  used  to  measure  the  position  of  the  vehicle  relative 
to  a lane  marker.  It  is  mounted  on  the  vehicle  and  points  to  the  side  and  downward  as  close  to 
perpendicular  as  possible.  Currently,  only  a single  camera  is  used  to  capture  either  the  inside  or 
outside  lane  marker.  There  are  several  methods  to  extend  the  BMS  to  multiple  cameras  so  that 
both  markers  can  be  detected.  The  camera  is  calibrated  so  that  image  coordinates  in  pixels  can 
be  transformed  into  road-surface  coordinates  in  millimeters. 

The  GPS/INS  subsystem  measures  vehicle  position  and  velocity  using  a carrier  phase  differential 
GPS  augmented  with  an  inertial  navigation  system.  When  GPS/INS  data  is  available,  the  camera 
can  be  calibrated  further  so  that  image  coordinates  can  be  transformed  into  GPS  coordinates 
(northing  and  easting).  It  should  be  noted  that  a warning  system  can  be  evaluated  without  the 
GPS/INS  subsystem.  The  camera  system  data  provides  the  distance  to  the  lane  marker,  which  is 
sufficient  for  determining  when  a vehicle  departs  a lane.  The  GPS/INS  subsystem  is  necessary 
when  the  evaluation  also  requires  where  the  vehicle  departs  the  lane.  Both  video  and  navigation 
data  are  stored  digitally  on  an  array  of  hard  disks.  Digital  storage  on  disk  enables 
synchronization  of  all  the  data.  In  addition,  the  data  can  be  examined  immediately  to  ensure  its 
quality.  Finally,  the  analysis  of  the  data  is  not  limited  by  the  onboard  processor’s  requirement  to 
respond  to  real-time  events. 
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The  third  component  shown  in  Figure  1 is  the  warning  system  under  evaluation.  An  interface  is 
developed  between  the  BMS  and  the  warning  system  so  that  roadway  departure  warnings 
generated  by  the  system  can  be  recorded  along  with  the  BMS  video  and  GPS/INS  data.  Other 
types  of  warning  system  data  may  be  recorded,  including: 

1 . Loss  of  lane  marker  tracking.  This  may  result  when  the  lane  markers  fade  or  disappear. 

2.  Time-to-departure  from  lane.  The  estimated  time  until  the  vehicle  crosses  the  lane  marker. 

3.  Distance  from  lane  marker  and  closing  rate  to  lane  marker. 

4.  Lane  curvature. 

2.2  Collecting  Roadway  Departure  Incident  Data 

The  purpose  of  the  data  collection  is  to  obtain  data  describing  the  performance  of  the  warning 
system  under  a wide  variety  of  operational  conditions.  The  conditions  are  selected  to  closely 
mimic  real  driving  environments.  The  data  provides  insight  into  the  accuracy  of  the  system  in 
detecting  a roadway  departure  and  in  rejecting  false  alarms.  Generating  incidents  for  a rear-end 
collision  avoidance  system  is  difficult,  but  this  is  not  the  case  for  a roadway  departure  warming 
system.  Driving  the  vehicle  over  various  roads  and  intentionally  crossing  over  the  lane  markers 
is  sufficient  for  a preliminary  evaluation. 

The  operational  parameters  for  data  collection  are  the  various  conditions  that  affect  the 
performance  of  the  warning  system.  Prior  to  the  evaluation,  each  parameter  is  identified  and  the 
space  the  parameter  spans  is  defined.  For  example,  lighting  is  one  parameter  and  it  spans  from 
full  daylight  to  nighttime.  When  possible,  the  parameter  is  measured  in  order  to  quantify  under 
what  conditions  the  data  was  collected.  The  following  is  a brief  list  of  some  parameters  that  may 
affect  system  performance. 

1.  Lane  proximity  - distance  between  the  vehicle  and  lane  marker. 

2.  Lane  departure  rate  - rate  at  which  vehicle  can  approach  lane  marker. 

3.  Lane  departure  frequency  - amount  of  time  between  lane  crossings. 

4.  Road  surface  types  - asphalt,  concrete,  etc. 

5.  Road  surface  quality  - cracks,  potholes,  etc. 

6.  Lane  marker  contrast  - bright,  faded,  etc. 

7.  Lane  marker  type  - solid,  stripped,  etc. 

8.  Lane  types  - merge,  split,  intersection,  etc. 

9.  Lane  curvature  - curvature  of  road  the  system  can  monitor. 

10.  Light  intensity  - direct  sunlight,  clouds,  dusk,  night,  etc. 

1 1.  Weather  condition.  Dry,  rain,  snow,  etc. 

Once  the  BMS  is  installed  on  the  vehicle  and  calibrated,  the  process  of  collecting  data  is 
straightforward.  First,  a time  and  place  is  selected  based  on  the  type  of  data  needed,  i.e., 
nighttime,  rainy  weather  on  interstate.  The  driver  turns  on  the  BMS  and  starts  driving.  The 
driver  performs  maneuvers  that  test  the  response  of  the  warning  system  to  both  lane  crossings 
and  non-lane  crossings.  The  data  is  collected  in  relatively  short  segments  of  30  to  60  minutes 
each.  During  each  data  collection,  the  driving  is  divided  into  two  phases.  In  the  first  phase,  no 
lane  crossings  are  performed  in  order  to  gather  data  characterizing  the  false  alarm  rate  of  the 
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warning  system.  In  the  second  phase,  a variety  of  maneuvers  are  performed  that  cause  the 
vehicle  to  cross  a lane  marker.  This  can  be  done  in  a safe  manner  by  performing  the  maneuver  on 
sections  of  road  where  no  other  vehicles  are  close  by.  Also,  run-off-road  maneuvers  can  be 
performed  along  roads  where  sufficient  asphalt  surface  exists  on  the  other  side  of  the  lane  marker 
to  permit  safe  control  of  the  vehicle  after  the  lane  marker  is  crossed. 

The  steps  performed  by  the  driver  to  collect  data  are  summarized  in  the  following  list. 

1 . Select  time  and  place  for  data  collection  in  order  to  satisfy  desired  operational  parameters. 

2.  Turn  on  BMS  recording  and  verify  data  quality. 

3.  Drive  down  center  of  lane  for  10  minutes. 

4.  Decrease  distance  to  lane  marker  in  increments  of  15  cm  (6  inches)  and  maintain  distance  for 
1 minute.  Stop  decreasing  distance  prior  to  contact  with  lane  marker.  Return  to  center  and 
perform  same  steps  toward  opposite  lane  marker  (assumes  BMS  records  lane  position  on 
both  sides).  Return  to  lane  center  when  done. 

5.  Slowly  oscillate  position  of  vehicle  within  lane  without  touching  lane  marker.  In  safe 
manner,  slowly  increase  frequency  of  oscillation. 

6.  Select  section  of  road  where  lane  crossings  can  be  performed. 

7.  Perform  several  lane  changes  in  both  directions. 

8.  Slowly  oscillate  position  of  vehicle  so  as  to  cross  lane  marker  repeatedly.  In  safe  manner, 
slowly  increase  frequency  of  oscillation. 

9.  Turn  off  BMS  recording  and  verify  data  quality. 

10.  Make  entries  describing  conditions  during  data  collection  not  automatically  recorded  by  the 
BMS  (weather,  road  quality,  etc.). 


After  the  collection  run  is  complete,  the  data  is  transferred  from  the  vehicle  to  a workstation 
where  warning  system  performance  is  evaluated  using  BMS  analysis  software. 


2.3  Performance  Analysis 

The  performance  of  the  warning  system  is  judged  by  its  reliability,  that  is,  the  ability  to  warn 
when  a lane  departure  takes  place  and  to  reject  false  alarms.  A quantitative  measure  of 
performance  (MOP)  for  reliability  of  the  warning  system  is  defined  as: 


general  reliability  = 


f successful  outcomes 
^ all  possible  outcomes 


\ 

•100  = 

J 


' TP  + 77V 
JP  + TN  + FP  + FN 


\ 

•100 

j 


(2-1) 


The  general  reliability  of  the  warning  system  is  obtained  by  comparing  the  output  of  the  warning 
system  against  the  data  collected  by  the  BMS.  In  each  video  frame,  both  the  BMS  and  the 
warning  system  measure  the  distance  between  the  vehicle  and  the  lane  marker.  For  a basic 
system  that  warns  when  the  lane  edge  is  crossed,  if  the  distance  goes  to  zero,  then  the  vehicle  has 
departed  the  road  and  the  warning  system  should  signal  a roadway  departure  incident.  Column  1 
in  Table  1 labels  the  possible  responses  of  the  warning  system.  The  remaining  entries  in  the  table 
describe  the  possible  outcomes  of  a comparison  between  the  warning  system  and  the  BMS  data. 
A true  positive  (TP)  outcome  indicates  that  both  the  warning  system  and  the  BMS  data  indicate  a 
roadway  departure  incident.  A false  positive  (FP)  outcome,  or  a false  alarm,  occurs  when  the 
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warning  system  issues  a warning  and  none  is  detected  in  the  BMS  data.  A false  negative 
outcome,  potentially  the  most  catastrophic  result,  occurs  when  the  warning  system  completely 
misses  a roadway  departure.  Finally,  a true  negative  indicates  that  the  warning  system  is  not 
responding  and  none  exists. 


Warning  System  Response 

BMS  indicates  roadway 
departure 

BMS  indicates  no  roadway 
departure 

warning  system  warns  of 
roadway  departure 

True  Positive  (TP) 
Correctly  warns  of  roadway 
departure 

False  Positive  (FP) 

False  alarm 

warning  system  does  not  warn 
of  roadway  departure 

False  Negative  (FN) 
Misses  roadway  departure 

True  Negative  (TN) 
Correctly  does  not  warn  of 
roadway  departure. 

Table  1.  The  possible  outcomes  of  a warning  system  evaluation  at  each  instance  in  time. 


The  general  reliability  metric  rewards  a warning  system  for  correctly  not  warning  of  a roadway 
departure  (true  negative).  This  leads  to  a problem  with  the  metric  because  the  majority  of  the 
data  will  produce  true  negative  outcomes.  This  bias  may  produce  a high  reliability  value  even 
when  the  warning  system  misses  all  roadway  departures.  For  example,  consider  the  following 
case  where  the  warning  system  misses  all  10  lane  departures  (TP  + FN  defines  the  total  number 
of  lane  departures)  and  yet  scores  a general  reliability  of  99%: 

TP  = 0 


77V  = 1000 


FP  = 0 
FN  = 10 


general  reliability  = 


' 0 + 1000 
^0  + 1000  + 0 + 10 


X 

• 100  = 99% 

J 


An  alternative  metric  focuses  on  the  reliability  of  the  warning  system  in  critical  situations  when 
the  vehicle  departs  the  road. 


critical  reliability  = 


' TP 
JP  + FN 


(2-2) 


A critical  reliability  of  100%  indicates  that  the  warning  system  never  failed  to  detect  a roadway 
departure  when  one  occurred.  Based  on  the  example  above,  the  critical  reliability  would  be  0% 
and  not  99%.  The  failure  rate  of  the  warning  system  can  be  derived  from  the  warning  system’s 
critical  reliability. 

( FN 


failure  rate  — 1 - critical  reliability 


(2-3) 


The  false  alarm  rate  of  the  warning  system  is  the  number  of  false  positives  over  some  number  of 
possible  outcomes.  The  question  is  over  what  range  of  possible  outcomes  is  the  rate  calculated? 

f FP 


false  alarm  rate  - 


TP  + TN  + FP+  FN 


•100 


(2-4) 


J 


Again,  including  a large  number  of  true  negative  outcomes  in  the  denominator  will  bias  the  false 
alarm  rate  toward  a smaller  value.  Since  the  false  alarm  rate  is  more  of  an  annoyance  than  a 
failure,  and  false  alarms  can  occur  at  any  time  (not  just  during  critical  moments),  the  rate  is 
defined  over  all  possible  outcomes. 

A report  is  generated  that  summarizes  the  results  of  the  analysis  for  all  sets  of  data.  The  report 
specifies  the  state  of  the  operational  parameters  listed  in  section  2.2  and  the  metrics  described 
above.  The  metrics  will  vary  for  each  set  of  data.  For  example,  the  reliability  may  be  98% 
during  daylight  hours  and  88%  at  night.  A cumulative  reliability  is  computed  based  on  the 
results  under  all  the  conditions. 


The  preceding  scenario  provides  one  example  of  how  the  BMS  is  used  to  evaluate  a roadway 
departure  warning  system.  Reliability  is  the  primary  measure  of  performance  but  additional 
measures  may  be  appropriate.  These  measures  will  be  developed  as  the  goals  of  the  evaluation 
of  a specific  warning  system  are  finalized.  The  BMS  is  a general-purpose  system  and  can  be 
programmed  to  meet  a wide  range  of  performance  analysis  requirements.  The  remaining 
sections  in  this  report  provide  details  of  the  design,  integration,  and  capability  of  the  BMS. 
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3 Baseline  Measurement  System 

The  BMS  is  primarily  a data  collection  system  designed  to  capture  video,  GPS  data,  and  the 
output  from  a countermeasure  system  under  evaluation.  A digital  recording  system  is  used  to 
capture  and  synchronize  all  the  data  more  easily.  Recording  the  data,  as  opposed  to  performing 
the  evaluation  in  real-time,  has  several  advantages.  First,  and  foremost,  is  the  ability  to  analyze 
the  data  in  order  to  reduce  the  uncertainty  of  the  reference.  Without  the  ability  to  analyze  the 
data,  it  is  difficult  to  make  the  assertion  that  the  BMS  serves  as  the  reference  for  other 
measurements.  As  an  example,  suppose  a roadway  departure  system  under  evaluation  reports 
that  the  vehicle  crossed  a lane  marker  at  a particular  position  or  instance  in  time.  The  BMS,  on 
the  other  hand,  reports  that  the  vehicle  did  not  cross  a lane  marker  at  that  point.  Which  system  is 
correct?  Recording  the  entire  event  enables  the  evaluator  to  examine  the  data  and  judge  which  is 
more  accurate.  With  the  visual  data  recorded,  additional  methods  can  be  used  to  improve  the 
uncertainty.  For  example,  survey  markers  can  be  used  as  a reference  to  improve  position 
measurements.  Another  advantage  of  recording  the  data  is  the  ability  to  conduct  a variety  of 
analyses  that  may  not  be  possible  under  real-time  constraints. 

The  BMS  consists  of  two  distinct  subsystems.  The  GPS/INS  subsystem  is  responsible  for 
measuring  vehicle  position  in  GPS  coordinates.  The  next  section  covers  the  Inertial  Navigation 
System  and  the  Differential  GPS  separately.  The  focus  of  the  descriptions  is  to  present  the 
operating  characteristics  and  the  accuracy  of  each  component  as  if  they  were  used  individually. 
The  section  on  the  Kalman  filter  describes  how  the  INS  and  GPS  measurements  are  combined 
together  to  achieve  a measurement  system  that  exhibits  the  best  characteristics  of  each. 

The  second  subsystem  of  the  BMS  is  the  video/data  digital  recording  system  (VDDRS).  The 
VDDRS  is  based  on  a ruggedized,  off-the-shelf,  Silicon  Graphics  Inc.  02  computer  system.  The 
02  records  camera  input  and  data  from  the  GPS/INS  subsystem  onto  an  array  of  disks.  The 
camera  is  calibrated  so  that  points  in  the  image  can  be  transformed  into  GPS  coordinates.  A 
variety  of  analysis  software  exists  that  automatically  locates  a lane  marker  in  a video  frame, 
detects  lane  crossings,  and  plots  the  results  of  an  evaluation.  The  camera  calibration  and  the 
digital  recorder  are  covered  in  the  remaining  portions  of  this  section. 


3.1  Inertial  Navigation  System 

The  U.S.  Army’s  Modular  Azimuth  and  Positioning  System,  MAPS2,  is  an  inertial  navigation 
system  containing  three  ring  laser  gyros,  three  accelerometers,  and  a rear  axle  odometer.  MAPS 
can  supply  orientation  and  translation  data  at  5 Hz.  A Navigation  Interface  Unit  (NIU), 
previously  developed  by  Alliant  Tech,  is  used  to  increase  the  update  rate.  The  NIU  requests 
orientation  data  from  the  MAPS,  which  is  available  by  itself  at  25  Hz,  and  taps  into  the  odometer 
to  measure  the  distance  traveled.  It  combines  the  odometer  and  heading  data  to  generate  vehicle 
position  data  at  25  Hz. 

The  MAPS/NIU  position  measurement  drifts  as  the  vehicle  moves,  due  mainly  to  wheel  slippage. 
Our  estimate  of  the  drift  rate  of  the  system  on  pavement  and  in  dry  grass  is  slightly  more  than 
0.1%  of  the  distance  traveled.  The  estimate  was  obtained  by  first  marking  on  the  pavement  the 
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initial  location  of  the  tires  and  recording  the  vehicle  position  reported  by  the  system.  The  vehicle 
was  driven  along  a course  and  returned  as  close  as  possible  to  the  initial  tire  locations.  The 
output  of  the  system  was  then  compared  to  the  initial  vehicle  position  and  the  difference  was 
calculated  to  be  0.1%  of  the  distance  traveled.  The  runs  were  not  repeated  enough  to  obtain  a 
useful  standard  deviation  of  the  estimate.  Because  the  system  uses  odometry  obtained  from  the 
wheels,  the  system  drifts  more  when  driving  on  slippery  surfaces  and  steep  slopes. 

Figure  2 shows  the  inertial  dead  reckoning  system  drifting  as  the  vehicle  is  driven  around  a 
square  track.  The  inertial  system  drifts  to  the  north  and  to  the  west  by  a similar  amount  each 
pass.  In  this  example,  the  vehicle  was  driven  through  patches  of  snow  causing  the  inertial  dead 
reckoning  system  to  drift  at  0.4%  of  distance  traveled. 


-20  0 20  40  60  80  100  120 

Easting  (m) 


Figure  2.  Comparison  between  inertial  sensor  drift  and  GPS. 

3.2  GPS 

The  satellite  based  Global  Positioning  System,  GPS,  has  a stated  accuracy"  of  100  m in  the  stand- 
alone mode  for  civilian  receivers  and  around  10  m to  30  m for  stand  alone  military  receivers.  To 
improve  accuracy,  differential  GPS  is  used.  Figure  3 shows  the  components  of  the  differential 
GPS  for  the  NIST  vehicle.  A stationary  receiver  at  a known  location,  in  this  case,  located  on  top 
of  NIST  Administration  Building,  transmits  perceived  position  errors  over  a data  radio  to  the 


" These  GPS  accuracy  figures  are  from  the  1994  Federal  Radionavigation  Plan.  The  figures  express  the  value  of  two 
standard  deviations  of  radial  error  from  the  actual  antenna  position  to  an  ensemble  of  position  estimates  made  under 
specified  satellite  conditions. 
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moving  GPS  receiver  onboard  the  vehicle.  The  vehicle  receiver  uses  this  data  to  improve  its 
position  estimate.  Using  this  technique,  accuracies  from  several  meters  down  to  sub-meter  can 
be  obtained.  Even  higher  accuracy  can  be  obtained  by  including  carrier  phase  measurements  in 
the  transmissions.  Using  long-term  averaging  and  post  processing,  carrier  phase  differential  GPS 
can  achieve  accuracies  of  a centimeter  or  better  while  the  receiver  is  stationary.  An  important 
feature  of  carrier  phase  differential  GPS,  available  within  the  last  two  years  on  commercial 
systems,  is  the  ability  to  resolve  the  cycle  integer  ambiguities  of  the  carrier  phase  while  the 
remote  receiver  is  moving.  This  allows  the  vehicle  to  lose  satellite  lock  (caused  by  driving  under 
a bridge,  near  buildings,  through  foliage,  etc.)  and  then  reacquire  phase  lock  on  the  fly.  When 
there  are  not  enough  satellites  visible  for  GPS  to  calculate  an  accurate  position,  the  GPS/INS 
system  relies  solely  on  the  inertial  dead  reckoning  of  the  MAPS  and  NIU. 


Figure  3.  Components  of  the  GPS  and  INS. 

3.3  Kalman  Filter 

Both  the  INS  and  the  GPS  systems  have  advantages  and  disadvantages.  INS  has  a higher  data 
rate  of  25  Hz,  but  it  drifts.  The  GPS  system  does  not  drift  significantly  over  time,  but  it  has  a 
noise  component,  a slower  data  rate  of  2 Hz,  and  is  not  always  available.  To  get  the  best  of  both 
systems,  their  outputs  are  combined  using  a complementary  filter,  shown  in  Figure  4. 
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Figure  4.  A complementary  filter  is  used  to  estimate  the  true  position  of  the  vehicle. 


The  vehicle  is  at  some  position  we  will  call  truth.  Both  navigation  sensors  try  to  measure  this 
but  have  some  inherent  error.  The  inertial  dead  reckoning  system  reports  truth  with  an  error  we 
will  call  drift , and  the  GPS  also  reports  truth  but  with  an  error  we  will  call  GPS  noise.  Taking 
the  difference  between  the  output  of  the  two  sensors  cancels  the  true  position  and  results  in  the 
difference  between  the  drift  and  the  GPS  noise.  The  Kalman  filter  tries  to  remove  the  noise  and 
estimate  the  drift.  Subtracting  this  drift  estimate  from  the  inertial  dead  reckoning  system  output 
yields  an  estimate  of  the  true  vehicle  position.  The  Kalman  filter  runs  once  for  each  GPS  update, 
producing  drift  estimates  at  2 Hz.  However,  truth  is  estimated  at  25  Hz  using  the  current  inertial 
dead  reckoning  data  and  latest  drift  estimate.  In  general,  drift  changes  slowly,  allowing  the 
calculation  of  position  between  GPS  updates  and  during  short  GPS  dropouts. 

The  Kalman  filter  uses  a simple  model  for  drift  that  assumes  the  drifts  in  Northing  and  in  Easting 
are  independent  of  each  other.  This  results  in  two  separate  scalar  filters.  Recall  that  the  filter  is 
estimating  drift  and  not  position.  The  state  and  output  model  is 

d , — d + w 
n + 1 n n 

Z =d  + v <«> 

n n n 

where 

dn  is  the  drift  at  time  n, 

wn  is  the  system  noise  with  covariance  g(I, 

z,x  is  the  measured  drift  (z„  - Inertial  - GPS),  and 

vn  is  the  measurement  noise  with  covariance  rn. 

The  system  noise  wn  models  wheel  slippage  and  other  factors  that  cause  the  drift  to  change.  As 
drift  tends  to  increase  with  the  distance  traveled,  the  covariance  of  the  system  noise,  qn,  is 
modeled  as  being  proportional  to  the  distance  traveled  since  the  last  GPS  update.  The 
measurement  noise  v is  the  GPS  noise.  Its  covariance,  r , is  modeled  as  a constant. 

n 1 /?7 
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The  Kalman  filter  reduces  to  the  following  scalar  equations  (one  set  for  Northing  and  an  identical 
set  for  Easting) 

Kn  = Pn/(Pn  +0 

dn+x=dn+Kn(zn-dn')  ( 3-2) 

Pn+l  = Pn(l~Kn)+<ln 


where 

Kn  is  the  Kalman  filter  gain,  and 

dn  is  the  estimated  drift  with  an  error  covariance  pn . 

The  recursive  filter  is  started  with  an  initial  guess  for  da  and  po. 

The  position  of  the  vehicle  is  then  estimated  as 

~ Truthn  = Inertial n —dn  ( 3-3  ) 


Figure  5.  The  Kalman  filter  computes  estimated  drift  and  noise  in  the  Northing  direction. 


Figure  5 show  the  results  of  the  Kalman  filter  for  the  first  three  laps  of  the  square  course  shown 
in  Figure  2.  The  GPS  receiver  is  operating  in  the  standard  (non-carrier  phase)  differential  mode. 
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which  provides  position  data  to  1 meter  (as  claimed  by  the  manufacturer).  Both  the  measured 
drift,  z,  and  the  estimated  drift,  d , increase  in  magnitude  with  time.  During  each  lap  the  vehicle 
drifts  in  a similar  direction  at  the  same  location  producing  a repetitive  pattern  about  every  100  s. 
The  sensor  noise,  i.e.,  the  GPS  noise,  cannot  be  determined  exactly,  but  it  can  be  estimated  as  (z 
- d ).  Note  the  peaks  at  50  s,  150  s,  and  250  s.  These  were  caused  when  the  vehicle  slipped  on  a 
snow  patch  causing  a large  change  in  drift  that  the  filter  erroneously  attributed  to  GPS  noise. 

This  mistake  is  corrected  in  a few  seconds  as  further  GPS  updates  corroborate  the  large  change  in 
drift. 

3.4  Camera  Calibration 

The  BMS  camera  is  used  to  make  measurements  of  features  on  the  road.  The  camera  calibration 
is  the  process  of  establishing  the  transformation  of  points  in  the  image  into  the  corresponding 
points  on  the  road  surface.  In  the  general  case,  camera  calibration  is  a difficult  problem.  The 
calibration  process  can  be  greatly  simplified  for  the  case  where  the  camera  is  fixed  on  the  vehicle 
and  is  pointed  relatively  straight  down  on  the  road  surface.  The  picture  in  Figure  6 shows  how 
the  camera  is  mounted  to  the  vehicle  for  the  tests  conducted  in  this  report. 


Figure  6.  Picture  of  downward  pointing  camera  mounted  on  side  of  vehicle. 

The  calibration  process  is  divided  into  two  phases.  In  the  first  phase,  a look-up  table  is  built  that 
defines  a transformation  from  points  in  the  image  plane,  expressed  in  pixel  coordinates,  into 
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points  on  a local  coordinate  system  on  the  road  surface,  expressed  in  millimeters.  In  the  second 
phase  of  calibration,  the  transformation  between  the  local  road-surface  coordinate  system  and  the 
GPS  coordinates  of  the  vehicle  is  established.  Once  the  calibration  is  completed,  a routine  can  be 
called  which  transforms  (i,j)  camera  points  into  (North,  East)  GPS  points. 

3.4.1  Local  Road-Surface  Calibration 

The  calibration  for  the  local  road  surface  establishes  a transformation  between  image  points  and 
road-surface  points.  A simple  calibration  is  a scale  factor  that  converts  pixels  into  millimeters. 

A single  scale  factor  does  not  accurately  model  the  transformation  from  image  coordinates  to 
world  coordinates.  A lookup  table  calibration  is  used  in  the  BMS  to  store  scale  factors 
throughout  the  image  plane.  The  calibration  is  good  for  a single  camera  pose  (position  and 
orientation)  relative  to  the  road  surface.  The  accuracy  of  the  calibration  depends  on  the  fact  that 
the  camera  pose  and  the  road-surface  orientation  do  not  vary  significantly.  Besides  camera  pose, 
other  variables  affect  the  calibration  process.  One  variable  is  the  lens.  A wide-angle  lens  will 
provide  a greater  coverage  but  will  decrease  the  resolution,  thereby  changing  the  number  of 
millimeters  per  pixel  (i.e.,  the  scale  factor).  A narrow-angle  lens  has  the  opposite  effect. 

Another  variable  is  the  image  format.  When  collecting  data  with  the  digital  recorder,  the  user 
can  select  the  number  of  rows  or  columns  of  an  image  to  save.  For  example,  in  order  to  save 
disk  space,  every  other  row  of  the  image  may  be  skipped.  In  a full  image,  the  480  pixels  in  a 
column  may  span  approximately  1.25  m on  the  road  surface.  Skipping  every  other  row  results  in 
240  pixels  spanning  1.25  m,  thereby  changing  the  calibration.  In  summary,  the  calibration  must 
be  performed  whenever  the  camera  pose,  lens,  or  image  digitization  format  is  changed. 

Figure  7 shows  a picture  of  the  target  used  to  calibrate  the  side-looking  camera.  The  target 
consists  of  a large,  flat  sheet  that  can  be  rolled  out  to  accommodate  varying  fields-of-view.  Since 
the  sheet  is  very  thin,  the  surface  beneath  is  also  effectively  calibrated  once  the  calibration  is 
complete.  The  calibration  target  has  a constant  offset  between  control  points,  but  the  camera 
pose,  lens,  and  image  format  distorts  the  offset.  An  image  processing  program,  written  in 
Khoros3,  computes  the  location  of  the  centroids  of  each  control  point  in  the  image.  Figure  7 
shows  the  output  of  the  program.  A white  boundary  around  the  perimeter  of  a control  point 
indicates  the  centroid  was  successfully  detected.  The  output  of  the  Khoros  program  is  an  ASCII 
file  containing  the  locations  of  the  centroid  of  each  control  point. 


i 


Figure  7.  Calibration  target  is  used  to  define  a local  road-surface  origin  (x,y)  at  top  left 
control  point. 

The  next  step  is  to  identify  the  control  points  in  the  image  that  correspond  with  the  control  points 
on  the  calibration  target.  A C program  is  used  for  this  step.  Figure  7 shows  the  two  coordinate 
systems  used  for  this  step.  The  origin  of  the  image  frame  is  at  the  top,  right  portion  of  the  image 
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(arrows  labeled  i and  j).  The  origin  of  a local  road-surface  coordinate  system  is  at  the  top,  right 
control  point  (arrows  labeled  x and  y ).  The  program  automatically  matches  centroids  in  the 
image  with  control  points  on  the  calibration  target.  It  then  assigns  offsets  in  the  x and  y 
directions  (relative  to  the  local  road-surface  coordinate  system)  to  each  centroid  based  on  the 
known  offsets  of  the  calibration  target. 

The  last  step  in  the  local  road-surface  calibration  is  to  build  a lookup  table  that  is  used  to 
transform  pixels  points  into  road-surface  points.  The  lookup  table,  a C data  structure,  is  created 
each  time  the  system  is  initialized.  Each  node  in  the  data  structure  contains  the  pixel  coordinates 
of  a control  point  in  the  image,  the  (x,y)  coordinates  of  the  corresponding  control  point  in  the 
local  road  surface  coordinate  system,  and  the  coefficients  for  interpolating  between  pixels  in  the 
horizontal  and  vertical  directions.  Once  the  lookup  table  is  created,  a lookup  routine  is  called 
whenever  there  is  a need  to  transform  an  image  point  into  a road-surface  point.  The  lookup 
routine  searches  the  data  structure  to  isolate  the  bounding  region  containing  the  pixel  and  then 
uses  linear  interpolation  to  transform  the  point  onto  the  road  surface. 

The  camera  calibration  technique  was  evaluated  to  determine  the  FOV  and  the  resolution  in 
transforming  image  points  to  road-surface  points.  Table  2 summarizes  certain  attributes  of  the 
camera  calibration  performed  for  this  report.  Cam  Ht  is  the  height  of  camera  (in  meters)  above 
the  road  (measured  by  hand).  Lens  is  the  lens  focal  length  (in  mm)  obtained  from  the  lens  body. 
Lat  (perpendicular  to  vehicle  travel)  and  Long  (parallel  to  vehicle  travel)  FOV  are  the  maximum 
coverage  of  the  camera  field-of-view  expressed  in  meters.  This  is  obtained  using  the  calibration 
lookup  table.  Max  and  Min  Res  are  the  distances  (in  mm)  on  the  road  surface  subtended  by  a 
pixel  in  the  lateral  direction  (along  the  x axis  in  Figure  7).  This  varies  with  pixel  location.  This 
is  obtained  using  the  lookup  table. 


Cam  Ht  m 

Lens  mm 

Lat  FOV  m 

Long  FOV  m 

Max  Res  mm 

Min  Res  mm 

1.375 

6 

1.7 

1.25 

1.86 

3.28 

Table  2.  Camera  calibration  attributes. 


3.4.2  Camera  to  GPS  Calibration 

To  complete  the  camera  calibration,  it  is  necessary  to  derive  the  transformation  from  the  local 
road-surface  coordinate  system  into  the  GPS  coordinate  system.  Figure  8 shows  some  relevant 
coordinate  systems  used  for  this  camera  calibration. 
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niu,  veh,  cam  and  gps_ant  are  rigidly  attached  to  vehicle, 
niu  and  veh  xy  planes  are  coplanar  with  axles, 
cam  xy  plane  is  coplanar  with  road. 

gps_world 


Figure  8.  Vehicle  coordinate  systems  describing  camera-to-GPS  calibration. 

The  BMS  obtains  the  location  of  the  vehicle  whenever  a camera  image  is  taken.  The  vehicle 
location,  shown  as  the  coordinate  system  (CS)  labeled  veh,  is  relative  to  the  gps_world  CS.  The 
camera-to-GPS  calibration  determines  the  transformation  from  the  veh  CS  to  the  cam  CS 
(labeled  veh-to-cam  transform  in  Figure  8).  The  x and  y axes  of  the  cam  CS  are  identical  to  the  x 
and  y axes  of  the  local  road-surface  CS  shown  in  Figure  7.  The  veh-to-cam  transformation 
depends  on  the  position  of  the  camera  on  the  vehicle  and  the  orientation  of  the  camera  with 
respect  to  the  vehicle  forward  axis.  Note  that  the  transformation  does  not  define  the  position  and 
orientation  of  the  camera  on  the  vehicle.  Once  the  veh-to-cam  transformation  is  known,  any 
pixel  in  the  camera  can  be  transformed  back  to  the  vehicle  and  then  into  GPS  coordinates. 

Computing  the  veh-to-cam  transformation  requires  that  two  points  on  the  road  be  surveyed 
manually  using  a GPS  antenna.  The  position  portion  of  the  transformation  only  requires  one 
point,  f,  shown  in  Figure  9. 
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Figure  9.  Derivation  of  position  transformation  between  the  local  road  surface  and  the 
GPS  coordinate  systems. 


Figure  9 shows  a top  view  of  the  coordinate  systems  shown  in  Figure  8.  Below  are  the 

definitions  of  the  systems. 

1.  gps  - the  world  CS  as  measured  by  the  orbiting  GPS  satellites. 

2.  veh  - a CS  centered  within  the  vehicle.  The  output  of  the  GPS  is  the  location  of  the  veh  CS 
with  respect  to  the  GPS  CS. 

3.  cam  - a CS  in  the  camera  field-of-view  whose  origin  is  the  local  road-surface  origin  shown  in 
Figure  7 rotated  so  the  x vector  in  the  vehicle  forward  direction.  The  output  of  the  lookup 
function  is  expressed  as  a vector  in  the  cam  CS. 

The  following  vectors  are  shown  in  Figure  9. 

1.  g_v  - the  position  of  the  vehicle  in  GPS  coordinates.  This  is  given  by  the  GPS. 

2.  c_f  - the  position  of  the  point  f in  the  cam  CS.  c_f  = lookup(ij)  where  (i,j)  is  the  pixel 
coordinates  of  point  f lookup  is  a routine  that  returns  the  GPS  coordinates  of  an  image  point 
(i,j).  The  point  is  selected  manually  while  viewing  the  image  on  a screen. 

3-  g_f  - the  position  of  point  f derived  by  manually  surveying  the  point  using  a GPS  antenna 
above  the  point. 

4.  v_c  - The  position  of  the  cam  CS  with  respect  to  the  veh  CS.  This  vector  is  the  translation 
portion  of  the  GPS  calibration,  which  is  unknown  and  is  the  value  to  derive. 

The  following  angles  are  also  shown  in  Figure  9: 

1.  0 - the  heading  of  the  vehicle.  This  is  derived  from  the  GPS  system  and  is  with  respect  to 
North. 

2.  cj)  - the  angle  of  the  cam  CS  with  respect  to  the  veh  x axis.  This  angle  is  the  orientation 
portion  of  the  GPS  calibration,  which  is  unknown  and  will  be  derived  later. 
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The  solution  for  the  veh-to-cam  transformation  does  not  require  the  roll  and  pitch  angles  of  the 
vehicle.  This  is  acceptable  since  the  roll  and  pitch  of  the  cam  CS  (i.e.,  the  local  road  surface) 
closely  parallels  the  veh  CS. 

As  can  be  seen  from  Figure  9,  the  vector  g_f  can  be  written  as: 

g_f  = trans(g_v)  + rotz($)  trans(v_c)  + rotz(6  + 0)  trans(c_f ) ( 3-4  ) 


[Note:  The  function  rotz  {angle)  defines  a rotation  of  angle  about  the  z axis  (positive  direction 
of  z is  out  from  the  page)  and  the  function  trans {vector)  defines  a translation  along  the  vector  (the 
magnitude  of  the  translation  is  equal  to  the  magnitude  of  the  vector).]  The  translation  vector  v_c 
can  be  derived  from  ( 3-4  ) as  follows: 

v_c  = rotz(-6)[  g_f  - rotz(6  + <p)  trans(c_f ) - trans(g_v)]  ( 3-5  ) 


The  orientation  portion  of  the  camera  calibration  does  not  require  the  roll  and  pitch,  only  the 
angle  between  the  forward  axes  of  the  cam  CS  and  the  veh  CS.  The  camera  orientation  can  be 
derived  using  points  b and  f in  Figure  10. 


Figure  10.  Two  points  are  required  to  derive  the  rotation  transformation  between  the  local 
road  surface  and  the  GPS  coordinate  systems. 


The  new  vectors  in  Figure  10  are: 
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1.  c_b  - the  position  of  the  point  b in  the  cam  CS.  c_b  = lookup(i.j)  where  (i,j)  is  the  pixel 
coordinates  of  point  b.  The  point  is  selected  manually  while  viewing  the  image  on  a screen. 

2.  g_b  - the  position  of  point  b derived  by  manually  surveying  the  point  using  a GPS  antenna 
above  the  point. 

3.  b_f  - the  vector  from  point  b to  point  f.  The  vector  can  be  derived  in  either  the  GPS  or  cam 
coordinate  systems. 

The  new  angle  in  Figure  10  is: 

1.  a - the  angle  of  the  cam  CS  with  respect  to  North.  This  angle,  as  shown  in  Figure  9,  equals  0 
plus  <\>. 

This  leads  to  the  following  equation  to  be  solved: 

0 = a - 6 ( 3-6 ) 

Figure  1 1 shows  how  the  difference  in  the  vectors  as  expressed  in  each  coordinate  system  is  used 

to  derive  the  rotation  portion  of  the  calibration. 


Figure  11.  Derivation  of  rotation  transformation  between  camera  and  GPS  coordinate 
systems. 


The  following  vectors  shown  in  Figure  1 1 can  be  derived  using  vector  addition  (see  also  Figure 
10): 


b_fcam  =c_f-c_b 
b_fgps  = g_f  - g_b 


(3-7) 


The  equations  in  ( 3-7  ) can  be  used  to  define  two  additional  angles: 
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1 . y - the  angle  of  b_fpps  with  respect  to  North. 

2.  X-  the  angle  of  b_fcam  with  respect  to  the  forward  axis  of  the  cam  CS. 
These  angles  are  derived  by: 

} = fl/2  - arctan(b_f  ) 

( 3-8  ) 

A = 7r/2  - arctan(b_f  cam) 


As  can  be  seen  in  Figure  1 1,  the  angle  X represents  the  angle  of  b_fcam  as  seen  through  the 
camera.  The  fact  this  differs  from  y,  the  angle  of  the  b_f  on  the  road,  is  the  result  of  the 
rotation  of  the  cam  coordinate  system  by  the  angle  a.  Therefore, 

j — ot  + A 

Given  the  known  values  y,  X,  0 and  equation 
(f)  = } - A -6 

= arctan(b_fcam ) - arctan(b_fgps ) - 6 


(3-9) 

( 3-6  ) yields: 

(3-10) 


Once  the  calibration  has  produced  the  position  and  orientation  parameters,  any  point  c_f  = 
lookup(i,j),  can  be  transformed  into  Northing  and  Easting  using  equation  ( 3-4  ). 

3.5  Video  Digital  Recording  System 

The  components  of  the  Video  Digital  Recording  System  are  shown  in  Figure  12.  They  consist  of 
a camera,  a Silicon  Graphics  Inc.  02  computer,  and  a RAID  (Redundant  Array  of  Inexpensive 
Disks)  storage  device.  A picture  of  the  system  is  shown  in  Figure  13.  The  02  is  used  to  digitize 
video,  read  GPS/INS  data,  read  data  from  a system  under  evaluation,  and  store  all  data  onto  the 
RAID.  The  RAID  consists  of  three  9-MB  hard  disks  that  operate  in  parallel  but  appear  as  a 
single  volume  in  the  file  system.  The  02  is  ruggedized  to  withstand  shock  and  vibration 
disturbances  when  the  vehicle  is  driven  off-road. 
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Figure  12.  Components  of  the  Video  Digital  Recording  System. 


Figure  13.  Picture  of  the  BMS  Video/Data  Digital  Recording  System. 


- 23  - 


Data  collection  consists  of  recording  video  and  GPS/INS  data  simultaneously.  A single  C 
routine  is  used  to  initialize  the  02  video  input,  to  read  in  each  image,  to  read  in  the  current 
GPS/INS  data,  to  format  the  image,  and  to  write  the  video  and  GPS/INS  data  to  disk. 

The  maximum  recording  bandwidth  of  the  02  is  achieved  by  routing  the  video  directly  to  a disk 
buffer  without  accessing  individual  pixels.  Any  processor  access  to  video  memory,  for  example 
to  examine  some  pixel  value,  significantly  reduces  the  recording  bandwidth.  The  02  is  able  to 
record  color  video  at  640x480  pixels  at  30  frames  per  second,  or  at  320x240  pixels  at  60  fields 
per  second.  Occasional  dropouts  may  occur.  The  color  format  uses  2 bytes  per  pixel.  The 
resultant  bandwidth  is  (640x480x2x30/ 1024A2)  • 17.6  Mb/s. 
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4 Evaluation  of  Baseline  Measurement  System 

Several  tests  were  conducted  to  evaluate  the  BMS  and  to  show  how  the  BMS  might  be  used  to 
measure  the  performance  of  a countermeasure  system.  Below  is  a summary  of  some  factors  that 
affect  the  accuracy  of  the  BMS.  Later  in  this  section  are  the  results  of  several  tests  conducted  to 
validate  the  BMS  and  to  evaluate  its  performance. 

GPS  Receiver 

Radio  link  dropout,  signal  multi-path,  satellite  geometry,  and  satellite-lock  (i.e.,  the  satellite  is  no 
longer  visible  due  to  trees,  building,  etc.)  all  affect  GPS  accuracy.  During  dynamic 
measurements  (i.e.,  while  the  vehicle  is  moving),  delays  in  processing  and  communications  are  a 
significant  source  of  error.  During  these  tests,  we  relied  on  the  real-time  capabilities  of  the  GPS 
without  any  corrections  or  post-processing  of  the  data. 

Inertial  Navigation  System 

In  order  to  increase  the  update  rate  of  the  vehicle  position  measurement,  only  the  INS  gyros  and 
odometry  are  used.  The  drift  in  this  mode  of  operation  is  primarily  due  to  wheel  slippage  and  is 
generally  a function  of  distance  traveled.  The  drift  is  reduced  whenever  the  GPS  updates  are 
received.  Thus,  the  predominate  effects  of  INS  drift  are  confined  to  the  periods  between  the  half 
second  GPS  updates  and  when  the  GPS  loses  lock  with  the  satellites. 

Kalman  Filter  Algorithm 

The  Kalman  Filter  combines  data  between  the  GPS  and  the  INS,  so  uncertainties  will  depend  to  a 
large  part  on  the  uncertainties  contributed  by  these  components.  In  addition,  a small  error  has 
been  identified  due  to  the  differences  in  the  latencies  of  the  GPS  and  INS.  An  algorithm  has 
been  developed  that  appears  to  correct  for  these  errors  but  the  algorithm  was  not  completely 
implemented  in  time  for  these  tests. 

Camera  Calibration 

The  camera  calibration  transforms  pixel  coordinates  into  GPS  coordinates.  There'  are  two  parts 
to  the  transformation.  The  first  part  transforms  pixels  coordinates  into  x,y  coordinates  (in  mm) 
in  a local  road-surface  coordinate  system  (i.e.,  the  road  in  the  camera  FOV).  The  uncertainties  in 
this  part  of  the  calibration  are  due  to  the  resolution  of  the  camera,  the  distortion  of  the  lens,  the 
camera  orientation  with  respect  to  the  road,  and  the  density  of  the  calibration  target.  The 
resolution  of  the  camera  falls  in  the  2 mm  to  3 mm  range.  The  accuracy  of  this  transformation  is 
difficult  to  isolate  due  to  the  large  number  of  factors,  but  it  can  be  assumed  the  uncertainty  is 
bounded  from  below  by  the  resolution.  The  second  part  of  the  transformation  converts  local 
road-surface  coordinates  into  GPS  coordinates.  Since  the  GPS  receivers  are  used  to  establish  this 
transformation,  the  accuracy  of  this  transformation  depends  on  the  uncertainties  of  the  GPS. 

Test  Procedures 

The  tests  consist  of  running  the  vehicle  over  various  parts  of  the  NIST  grounds  and  collecting 
data  using  the  BMS.  The  data  for  each  run  consists  of  video  recorded  at  60  Hz.  and,  for  each 
video  frame,  the  corresponding  data  from  the  GPS,  INS  and  the  Kalman  filter.  The  video  data 
and  the  position  (or  navigation)  data  are  stored  in  individual  files.  A naming  convention 
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simplifies  keeping  track  of  the  video  and  data.  Table  3 shows  the  video  file  names.  The  file 
name  contains  the  date  and  time  the  video  was  collected,  the  format  (in  this  case  60  Hz  field 
video)  and  the  dimensions  of  each  frame.  The  names  of  the  position  data  files  corresponding  to 
the  video  are  shown  in  Table  4.  By  looking  at  the  “time”  field  of  the  file  name,  one  can  see  that 
the  total  time  required  to  collect  six  runs  of  data  was  approximately  13  min  (from  hr-mm-ss  of 
10-39-30  to  10-52-56).  This  illustrates  the  ease  in  which  data  can  be  collected  using  the  BMS. 


Size  (bytes) 

File  Name  (_date_time_hzxformat_colsxrow) 

196,455,600 

_04-03-98_  1 0-39-30_60xFI_720x  1 2 1 .raw 

647,388,720 

_04-03-98_  1 0-4 1 -40_60xFI_7 20x  1 2 1 .raw 

621,078,480 

_04-03-98_  1 0-44-28_60xFI_720x  1 2 1 .raw 

664,638,480 

_04-03-98_  1 0-47 -46_60xFI_7 20x  1 2 1 .raw 

616,809,600 

_04-03-98_  1 0-50- 1 7_60xFI_720x  1 2 1 .raw 

1,259,493,840 

_04-03-98_  1 0-52-56_60xFI_720x  1 2 1 .raw 

Table  3.  Video  files  recorded  for  test. 


Size  (bytes) 

File  Name  (_date_time_hzxformat_colsxrow) 

505120 

_04-03-98_10-39-30.nav 

1664544 

_04-03-98_  1 0-4 1 -40.nav 

1596896 

_04-03-98_10-44-28.nav 

1708896 

_04-03-98_10-47-46.nav 

1585920 

_04-03-98_10-50-17.nav 

3238368 

_04-03-98_10-52-56.nav 

Table  4.  Navigation  data  files  corresponding  to  video  files  in  Table  3. 


A reference  for  comparison  was  established  by  inserting  nails  into  the  road  asphalt.  The  nails 
were  surveyed  manually  using  the  GPS.  The  survey  involved  long-term  averaging  and  the  use  of 
carrier  phase  corrections.  The  uncertainties  are  believed  to  lie  in  the  1 cm  to  2 cm  range.  The 
nails  were  highlighted,  see  Figure  14,  with  black  and  white  paint.  The  highlighted  nails  make  it 
easy  for  the  analyst  to  select  the  nail  in  the  image.  The  nails  provide  a reference  point  for 
evaluating  the  BMS.  At  any  video  frame  where  the  nail  is  visible,  the  nail  can  be  selected  and 
transformed  into  GPS  space.  The  coordinates  of  the  nail  acquired  visually  can  be  compared  to 
the  manual  survey  measurements  to  determine  uncertainties  in  the  BMS.  The  remainder  of  this 
section  covers  the  tests  and  the  results. 


Figure  14.  A single  frame  recorded  by  BMS  showing  two  survey  nails  highlighted  using 
black  and  white  paint. 
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4.1  Verification  of  Camera  Calibration 

The  goal  of  this  test  was  to  establish  the  correctness  of  the  camera  calibration  software.  The 
vehicle  was  driven  to  a point  where  a survey  nail  was  visible  in  the  video.  The  BMS  recorder 
was  turned  on  and  the  vehicle  was  driven  through  a half  figure  eight  and  stopped  facing  in  the 
opposite  direction  with  the  survey  nail  again  visible  in  the  video.  Figure  15  shows  the  trajectory 
of  the  vehicle  and  the  locations  of  the  nail  acquired  visually  using  the  BMS.  The  unfiltered_gps 
line  is  the  raw  GPS  data  at  half-second  updates,  the  filtered_gps  is  the  output  of  the  Kalman 
filter,  and  the  ins  is  the  output  of  the  MAPS. 


Half  Figure  8 - verify  nail  locations 

(script:  /home/timon/proj/4-3-98/graphHalf F i g 8 l.tcl) 
(data:  04-03-98 _ 1 0-39-30 . n a v ) 
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Figure  15.  Vehicle  driven  along  a half  figure-eight  trajectory  along  with  survey  nails 
acquired  using  video  at  beginning  and  end  (see  also  Figure  16). 


Acquiring  the  nail  in  both  direction  exercises  the  calibration  transformations  from  the  camera’s 
FOV  to  GPS  space.  If  the  calibration  parameters  and  the  transformations  are  correct,  the  nail 
should  align  at  the  beginning  and  at  the  end,  and  should  coincide  with  the  surveyed  coordinates 
of  the  nail.  Figure  16,  a zoomed  in  view  of  the  nails  in  Figure  15,  shows  the  coordinates  of  the 
nail  acquired  in  the  first  (legend  titles  ending  in  1)  and  last  (legend  titles  ending  in  2)  video 
frame.  The  coordinates  of  the  nail  are  calculated  using  the  vehicle’s  filtered  and  unfiltered  GPS 
coordinates.  The  origin  of  the  graph  is  the  surveyed  location  of  the  nail.  Two  of  the  points  align 
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very  closely  with  the  origin.  The  locations  of  the  nails  and  their  distance  from  the  surveyed  nails 
are  listed  in  Table  5. 


Half  Figure  8 - nails  only 

(script:  /home/timon/proj/4-3-98/graphHalfFig8_2.tcl) 
(data:  _04-03-98_1 0-39-30.  nav) 
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Figure  16.  Filtered  and  unfiltered  (raw)  GPS  coordinates  of  nail  at  beginning  and  end  of 
half  figure  8 trajectory. 


Data  Point  Name 

North  (m) 

East  (m) 

Distance  From  Origin  (cm) 

filNailN  1 

-0.096 

0.033 

10.2 

filNailN2 

0.005 

-0.023 

2.4 

unfilNailN  1 

0.021 

-0.011 

2.4 

unfilNailN2 

-0.187 

-0.023 

18.8 

Table  5.  Values  of  visually  acquired  nails  and  their  distance  from  the  origin  (i.e.,  the 
known  survey  location  of  the  nail). 

4.2  Static  Measurements 

This  test  seeks  to  establish  the  uncertainty  of  the  BMS  in  measuring  the  position  of  a survey  nail 
using  the  video  system  and  the  GPS  while  the  vehicle  is  static  (i.e.,  stationary).  Several  runs 
were  made  on  the  NIST  grounds.  Figure  17  shows  the  typical  path  in  GPS  space  the  vehicle  took 
during  a run. 
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n_vs_e 

South  Drive 


Figure  17.  Trajectory  of  vehicle  during  test  to  acquire  nail  at  beginning  and  end  of  run. 

The  survey  nail  was  acquired  visually  at  the  beginning  and  end  of  each  run.  The  location  of  the 
nail  was  obtained  after  the  vehicle  was  stopped  and  in  only  one  video  frame.  Figure  18  and 
Table  6 show  the  measured  locations  of  the  nail.  Only  runs  where  satellite-lock  was  maintained 
are  included  in  this  test.  Again,  the  origin  is  the  surveyed  location  of  the  nail.  The  coordinates 
of  the  visually  surveyed  nail  are  computed  using  the  unfiltered  (raw)  GPS  data  in  order  to  isolate 
the  errors  due  to  the  INS  and  the  Kalman  filter.  The  majority  of  the  points  are  close  to  the  origin 
It  is  unknown  what  caused  one  nail  to  appear  53.905  cm  from  the  origin.  The  standard 
uncertainty  (i.e.,  the  standard  deviation)  over  all  the  measurements  in  Table  6 is  +/-  18  cm.  The 
expanded  uncertainty4  (based  upon  a t-distribution  and  95%  confidence  level)  is  +/-  42  cm.  This 
is  within  the  stated  specifications  of  the  manufacturer" 
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Start/End  Nail  Location  using  straight  GPS  without  dropout 


(script:  /home/timon/proj/4-3-98/graphAIINailsOK.tcl) 


o 


o 


— T~ 

0 . 2 


O u_04_03_98 ,1 0_39_30_0 
O u 04_03_98_1 0_39_30_2254 
O u_0403_98_10  41 _40_0 
O u_04_03_98_10_41_40_7430 
O u_04_03_98_1 0_47^  46  _0 
O u„04_03  ^98_1 0_47_46_7628 
O u_04_03_98_1 0_50_1  7 _ 0 
O u„04_03_98_10_50_1 77079 


e (meters) 


Figure  18.  Results  of  visually  acquiring  survey  nail  at  the  beginning  and  end  of  several 
runs. 


Data  Point  Name 

North  (m) 

East  (m) 

Distance  From  Origin  (cm) 

u 04  03  98  10  39  30  0 

0.012 

-0.014 

1.822 

u_04_03_98_ 1 0_39_30_2254 

-0.187 

-0.024 

18.887 

u_04_03_98_ 1 0_41_40_0 

-0.537 

0.052 

53.905 

u 04  03  98  10  41  40  7430 

0.026 

-0.013 

2.870 

u_04_03_98_ 1 0_47_46_0 

0.050 

-0.025 

5.593 

u 04  03  98  10  47  46  7628 

0.041 

-0.028 

4.946 

u 04  03  98  10  50  17  0 

0.040 

-0.029 

4.905 

u_04_03_98_ 1 0_50_ 1 7_7079 

0.030 

-0.033 

4.481 

Table  6.  Visual-based  measurements  of  nail  (see  Figure  18)  with  distance  from  manual 
surveyed  coordinates  of  the  nail  (origin). 


4.3  Dynamic  Measurements  and  Latency  Calculation 

A dynamic  GPS  measurement,  as  opposed  to  a static  measurement,  is  obtained  while  the  vehicle 
is  moving.  Latency  in  obtaining  the  GPS  measurement  from  the  receiver  is  one  source  of 
uncertainty  when  making  dynamic  measurements.  An  approach  to  determine  latency  is  to 
compare  the  coordinates  of  a survey  nail  obtained  visually  while  driving  against  the  coordinates 
of  the  survey  nail  obtained  manually  by  survey. 

To  obtain  a dynamic  measurement,  a survey  nail  was  visually  captured  using  the  camera  while 
the  vehicle  was  driven  along  a road.  Figure  19  shows  only  the  section  of  the  vehicle’s  trajectory 
as  it  drives  by  the  surveyed  nail.  The  vehicle’s  trajectory,  the  circles  connected  by  lines,  is 
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labeled  as  filtered_gps.  Each  circle  represents  a Kalman  filter  update.  The  manually  surveyed 
coordinates  of  the  nail,  the  plus  sign  labeled  surveyedNail,  is  shown  at  the  origin.  The  nail 
acquired  visually  using  the  camera  and  GPS  filtered  data  is  shown  as  the  circle  labeled 

videoNail. 

Half  Figure  8 - verify  nail  locations 

(script:  /home/timon/proj/4-3-98/graphNailDelay.tcl) 

(data:  04-03-98  1 0 - 4 1 -4 0 . n a v ) 
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Figure  19.  The  error  due  to  latency  is  shown  in  the  discrepancy  between  the  coordinates  of 
the  surveyed  nail  (surveyedNail)  and  the  visually  acquired  nail  (videoNail)  as  the  vehicle 
(filtered_gps)  drives  by  at  10.75  m/s. 


Two  methods  are  used  to  calculate  the  latency.  One  method  uses  the  relationship  velocity  = 
distance  / time.  Let  distance  equal  the  difference  between  the  nail’s  surveyed  coordinates  and  the 
nail’s  coordinates  measured  visually.  Let  velocity  equal  the  speed  given  by  the  odometer  (10.75 
m/s).  Solving  for  time  in  the  velocity  relationship  yields  a latency  of  3 16  ms. 

To  calculate  the  latency  in  a different  manner,  each  point  along  the  vehicle’s  trajectory  is 
examined  and  the  point  falling  closest  to  the  survey  nail  is  identified.  For  this  data  set,  the 
vehicle  passed  closest  to  the  nail’s  coordinates  at  GPS  data  record  837  (this  data  record 
corresponds  with  video  frame  837,  which  is  labeled  “start”  in  Figure  19).  The  nail  was  visually 
located  in  video  frame  852  (this  corresponds  to  GPS  data  record  852).  Each  record  is  time 
stamped  and  subtracting  the  differences  between  the  time  stamps  yields  a latency  of  300  ms. 

This  compares  closely  with  the  previously  mentioned  latency  estimate  of  316  ms.  These 
methods  show  how  a latency  calculation  might  be  made  but  additional  measurements  need  to  be 
made  prior  to  obtaining  an  uncertainty  value  for  the  estimate. 

The  GPS  receiver  manufacturer  claims  their  fast  algorithm  for  solving  carrier  phase  GPS  location 
produces  a solution  in  less  than  250  ms.  The  additional  latency  measured  also  takes  into  account 
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the  delays  in  communications  between  the  BMS  and  the  GPS  receiver,  and  any  internal 
processing  in  the  BMS. 

A 300  ms  latency  increases  the  uncertainty  in  the  vehicle’s  position  proportional  to  velocity.  For 
example,  at  30  m/s  (roughly  60  miles/hr),  the  latency  increases  the  uncertainty  by  9 meters.  For 
longitudinal  measurements,  this  uncertainty  is  significant.  For  lateral  measurements,  since  the 
velocity  approaching  the  lane  marker  is  small,  the  uncertainty  is  smaller.  One  approach  to  reduce 
the  uncertainty,  not  examined  for  this  report,  is  by  extrapolating  vehicle  position  based  on  its 
previous  trajectory.  This  partially  depends  on  a small  variance  in  latency,  which  again  suggests 
additional  measurements  need  to  be  made. 


4.4  Affects  of  GPS  Dropout 

One  of  the  main  sources  of  GPS  errors  is  the  loss  of  satellite-lock.  During  two  of  the  test  runs, 
satellite  loss  occurred  and  the  BMS  relied  on  the  INS  for  position  measurement.  Figure  20 
shows  a run  starting  at  the  intersection  of  East  and  South  Dr.  on  the  NIST  grounds.  The  vehicle 
was  driven  onto  Muddy  Branch  Rd.,  back  into  NIST,  along  Center  Dr.,  and  returned  to  the  start 
location.  To  simplify  reading  the  axis  scales,  the  origin  of  the  graph  is  offset  from  absolute  GPS 
coordinates  to  the  start  location  of  the  vehicle.  The  run  took  approximately  4.5  min  and  covered 
a distance  of  2885  m. 

Example  of  gps  dropout 


0 


-1000 


(script:  /home /timon/proj/4 -3-98/graph Dropout3.tc I ) 
(data:  _04-03-98_1  0-52-56.  nav) 


-500  0 500 

e (meters) 


Figure  20.  A run  out  the  NIST  gate  onto  Muddy  Branch  Rd  and  then  back  to  the  start 
showing  mode  changes  in  the  navigation  system  (DR  = DeadReckoning,  DGPS  = 
Differential  GPS,  cpDGPS  = carrier  phase  DGPS). 
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The  circles  indicate  where  the  navigation  system  changed  modes.  Each  circle  is  labeled  with  the 
new  mode  and  the  video  frame  number  where  the  mode  change  occurred.  At  the  start  of  the  run, 
the  navigation  system  was  in  carrier  phase  DGPS  mode.  This  mode  is  sustained  until  the  vehicle 
was  driven  beneath  trees  (at  frame  9159)  and  satellite-lock  was  lost.  At  this  point,  the  navigation 
system  mode  changed  to  dead  reckoning  (DR)  and  the  system  relies  on  the  INS  for  position 
measurement.  Satellite  lock  was  obtained  a short  distance  later  (frame  9495)  but  lost  again 
(frame  9813).  At  frame  10632  the  satellites  were  reacquired  and  carrier  phase  was  established 
near  the  end  of  the  run  at  frame  13139.  The  plot  and  mode  changes  in  Figure  20  were  generated 
automatically  using  the  BMS  analysis  software. 

Table  7 shows  the  elapsed  time  between  navigation  system  mode  changes.  It  took  5 seconds  and 
14  seconds  to  acquire  satellites  (time  elapsed  between  DR  and  DGPS  modes).  The  longer  time 
reflects  the  additional  time  required  to  clear  trees  and  regain  satellite  line-of-site.  Once  satellite- 
lock  was  acquired,  it  took  44  seconds  (time  elapsed  between  DGPS  and  cpDGPS  modes)  to 
resolve  the  cycle  integer  ambiguity  required  for  carrier  phase  operation.  This  agrees  with  the 
manufacturer’s  claim  of  less  than  a minute. 


elapsed  (sec) 

new  mode 

frames 

0 

cpDGPS 

0 

163 

DR 

9159 

5 

DGPS 

9495 

5 

DR 

9813 

14 

DGPS 

10632 

44 

cpDGPS 

13139 

Table  7.  Elapsed  time  between  navigation  system  mode  changes. 
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Figure  21  is  an  enlarged  view  of  the  vehicle  trajectory  during  section  “a”  in  Figure  20.  The 
trajectory  appears  distorted  because  the  axes  scales  are  different  in  order  to  maximize  the  range 
of  the  plot.  The  figure  shows  how  the  output  of  the  Kalman  filter  tracks  the  INS,  offset  by  the 
previous  drift,  when  GPS  lock  is  lost.  The  drift  value  remains  constant  while  the  GPS  lock  is 
lost.  Each  circle  in  the  figure  represents  a GPS  update. 


Example  of  gps  dropout 

(script:  /home/timon/proj/4-3-98/graphDropout4.tcl) 
(data:  _04 - 03-9 8_1 0-52-56 . n a v) 


o unf  i ltered_gps 

" “ filtered  gps 
ins 

O 9094 
O 9159 
O 9495 
O 9813 
O 10632 


Figure  21.  Vehicle  position  updates  during  section  of  trajectory  labeled  “a”  in  Figure  20. 
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Figure  22  is  an  enlarged  view  of  the  vehicle  trajectory  during  section  “b”  in  Figure  20.  The 
figure  shows  how  the  output  of  the  Kalman  filter  converges  to  the  GPS  data  once  GPS  satellite 
lock  is  obtained. 

Example  of  gps  dropout 


(script:  /home/timon/proj/4-3-98/graphDropout5.tcl) 
(data:  _04-03-98_1  0-52-56.  nav) 
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Figure  22.  Vehicle  position  updates  during  section  of  trajectory  labeled  “b”  in  Figure  20. 
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Figure  23  is  an  enlarged  view  of  the  vehicle  trajectory  during  section  “c”  in  Figure  20.  The 
figure  shows  the  position  along  the  trajectory  when  the  receiver  achieved  carrier  phase  lock.  The 
Kalman  filter  converges  even  further  when  the  vehicle  slows  to  a stop. 


Example  of  gps  dropout 

(script:  /home/timon/proj/4-3-98/graphDropout6.tcl) 
(data:  04-03-98  10-52-56. nav) 
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Figure  23.  Vehicle  position  updates  during  section  of  trajectory  labeled  “c”  in  Figure  20. 
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Finally,  in  Figure  24,  the  locations  of  the  nail  have  been  picked  out  from  the  video  and 
transformed  into  GPS  space.  The  origin  of  the  graph  is  the  manually  surveyed  location  of  the 
nail.  The  circles  are  the  locations  of  the  nail  in  the  first  and  last  (14456)  video  frame  using  the 
unfiltered  (raw)  GPS  data.  The  pluses  are  the  nails  based  on  Kalman  filtered  output.  Both  sets 
of  data  are  within  10  centimeters  of  the  nail. 


All  Nails 

(script:  /home/timon/proj/4-3-98/graph10-52-56Nails.tcl) 
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Figure  24.  Nail  locations,  derived  from  navigation  system  (filtered_gps)  and  raw  GPS 
(unfiltered_gps)  data,  at  the  beginning  and  end  of  run  in  Figure  20 
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4.5  Lane  Position  Measurements 

This  test  shows  how  the  BMS  can  measure  the  location  of  the  passenger-side  lane  marker  while 
the  vehicle  is  moving.  A sequence  of  the  recorded  video  data  is  processed  and  the  lane  location 
in  each  frame  is  extracted  automatically  using  an  adjustable  template  correlation  function 
developed  at  Carnegie  Mellon  University  ’.  The  lane  location  is  transformed  into  absolute  GPS 
space.  The  lane  and  the  vehicle  locations  are  shown  in  Figure  25.  The  analysis  software 
automatically  labels  the  location  of  the  minimum  and  maximum  distances  between  the  vehicle 
and  the  lane.  The  software  can  be  extended  to  also  flag  when  the  vehicle  crosses  the  lane. 


T rack  Lane 

(script:  /home/timon/proj/4-3-98/graphTrackLane.tcl) 
(data:  ) 


i 1 I 1 I 1 I 

308740  308760  308780  308800 

e (meters) 


Figure  25.  Trajectory  of  vehicle  and  location  of  outside  lane  marker  shown  with  minimum 
and  maximum  distance  between  vehicle  and  lane  labeled  (min/max:  distance  from  lane  in 
meters,  GPS  record  number). 
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5 Conclusion 


The  thrust  of  the  AUTONAV  / DOT  project  has  been  to  develop  a quantitative  means  for 
measuring  the  performance  of  roadway  departure  warning  systems.  A series  of  performance 
measures  describing  the  reliability  of  a warning  system  has  been  defined.  A baseline 
measurement  system  (BMS)  has  been  developed  to  gather,  process  and  analyze  data  that  is  then 
used  to  compute  the  performance  measures. 

The  BMS  uses  a carrier  phase  differential  GPS  receiver  and  an  inertial  navigation  system  (ring 
laser  gyros  and  odometry)  to  measure  vehicle  position  and  a calibrated  camera  system  to  measure 
the  vehicle’s  position  relative  to  a lane  marker.  An  interface  is  developed  between  the  BMS  and 
the  warning  system  so  that  the  output  of  the  warning  system  can  also  be  recorded.  The  output  of 
the  warning  system,  the  camera  and  GPS  is  synchronized  and  stored  digitally  onto  a large  array 
of  disks  in  real-time  (60  Hz)  during  normal  driving  scenarios.  Afterwards,  the  data  is  analyzed  to 
determine  when  the  vehicle  departs  a lane.  The  time  of  departure  is  compared  against  the  output 
of  the  warning  system.  True  positive  and  true  negative  outcomes  indicate  reliable  operation, 
while  false  positive  and  false  negative  outcomes  indicate  unreliable  operation.  The  tally  of  the 
outcomes  is  used  to  produce  reliability,  failure  and  false  alarm  rates,  which  are  the  primary 
performance  metrics. 


-39- 


Appendix  A:  Autonav  Program  Description 

The  goal  of  the  U.S.  Department  of  Defense,  AUTONAV  (Autonomous  Vehicle  Navigation 
Control  System)  project  is  to  develop  a control  system  and  sensor  suite  that  when  mounted  on  a 
vehicle  will  enable  the  vehicle  to  autonomously  navigate.  The  vehicles  will  be  used  to  perform 
scouting  and  reconnaissance  missions  without  putting  the  soldier  in  danger.  The  project  team  is 
integrating  and  evaluating  several  sensor  and  control  system  components  that  may  be  of  use  to 
the  Department  of  Transportation.  Below  is  a summary  of  the  main  components. 


UBM  4D  Perception  and  Control  System 

The  UBM  (Universitat  der  Bundeswehr  Munchen)  4D  Perception  and  Control  System7  is  a 
system  capable  of  a wide  range  of  autonomous  driving  functions.  The  UBM  system  has 
demonstrated  the  ability  to  drive  autonomously  along  a highway,  to  perceive  traffic  in  front  and 
to  the  sides,  to  maintain  speed  and  spacing  with  a vehicle  in  front,  and  to  pass  vehicles  when 
appropriate.  A core  component  of  the  UBM  system  is  the  Road  Detection  and  Tracking  (RDT) 
module.  The  RDT  is  capable  of  looking  forward,  detecting  lane  markers,  tracking  the  lane 
markers,  and  computing  lane  curvatures.  The  system  is  currently  undergoing  conversion  to  run 
on  standard  PC  hardware  and  will  be  integrated  on  the  NIST  vehicle  and  evaluated  in  the  1999 
time  frame. 

DSRC  Vision  Front  End 

The  David  Sarnoff  Research  Center  VFES  is  a special-purpose  vision  processor  that  can  perform 
complex  operations  on  video  in  real-time.  Two  of  the  primary  functions  of  the  VFE  for 
AUTONAV  is  image  stabilization  and  stereo  processing.  Image  stabilization  is  done 
electronically  by  detecting  global  motion  in  the  image  and  then  warping  the  image  to  subtract  out 
the  motion.  Stabilization  is  an  important  preprocessing  step  to  any  countermeasure  system  based 
on  video.  Stabilization  helps  simplify  image  processing  algorithms  and  improves  accuracy. 
Stereo  processing  takes  images  from  two  cameras  and  produces  a single  range  image.  Each  pixel 
in  the  image  is  the  range  along  the  two  cameras’  view  axis.  Range  images  form  the  basis  for 
collision  detection  and  countermeasure  systems.  The  system  will  be  integrated  on  the  vehicle 
and  tested  in  the  1999  time  frame. 

EBK  LADAR 

EBK4  (Entfernungs-Bild-Kamera)  LADAR,  manufactured  by  Dornier,  uses  a laser  and  a two 
degree-of-freedom  scanning  mechanism  to  collect  range  measurements.  The  ladar  produces 
128x64  (horizontal  x vertical  pixels)  range  image  once  each  second,  though  each  pixel  is 
available  immediately  upon  measurement.  The  maximum  range  is  50  m and  the  range  resolution 
is  6 cm.  The  ladar  produces  range  images  similar  to  a stereo  system  but  with  different 
characteristics.  The  primary  drawback  is  the  time  it  takes  to  scan  the  laser  mechanically  and  to 
make  the  measurements.  The  advantage  of  ladar  over  stereo  is  the  ability  to  produce  more  robust 
readings  under  a wider  variety  of  conditions.  Tests  are  now  being  conducted  in  the  AUTONAV 
project  to  determine  the  performance  of  stereo  systems  versus  ladar.  NIST  is  also  looking  at 
flash  ladar  systems  that  do  not  require  a mechanically  scanned  laser.  A flash  ladar  illuminates 
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the  entire  scene  and  measures  range  simultaneously  throughout  the  image.  Typical  flash  ladars 
currently  under  development  have  the  ability  to  produce  up  to  a 320x240  range  image  at  10  Hz. 
The  range  images  from  a ladar  form  the  basis  for  collision  detection  and  collision  avoidance 
countermeasure  systems. 
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